Nighttime Sensing

ABSTRACT

Systems and methods for night vision combining sensor image types. Some implementations may include obtaining a long wave infrared image from a long wave infrared sensor; detecting an object in the long wave infrared image; identifying a region of interest associated with the object; adjusting a control parameter of a near infrared sensor based on data associated with the region of interest; obtaining a near infrared image captured using the adjusted control parameter of the near infrared sensor; and determining a classification of the object based on data of the near infrared image associated with the region of interest.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.62/564,654, filed on Sep. 28, 2017, the content of which is herebyincorporated by reference in its entirety for all purposes.

TECHNICAL FIELD

This disclosure relates to multi-modal sensing for nighttime autonomousobject detection and recognition.

BACKGROUND

Some automated systems gather process large quantities of sensor data toidentify objects in the surrounding environment. The processing ofsensor data is often subject to a real-time constraint to facilitatenavigation and/or robust control of the automated system.

SUMMARY

Disclosed herein are implementations of multi-modal sensing fornighttime autonomous object detection and recognition.

In a first aspect, the subject matter described in this specificationcan be embodied in systems that include a body; actuators operable tocause motion of the body; a long wave infrared sensor; and a nearinfrared sensor. The systems include a processing apparatus configuredto obtain a long wave infrared image from the long wave infrared sensor,detect an object in the long wave infrared image, identify a region ofinterest associated with the object, adjust a control parameter of thenear infrared sensor based on data associated with the region ofinterest, obtain a near infrared image captured using the adjustedcontrol parameter of the near infrared sensor, determine aclassification of the object based on data of the near infrared imageassociated with the region of interest, determine a motion plan based onthe classification of the object, and output commands to the actuatorsto maneuver the system.

In a second aspect, the subject matter described in this specificationcan be embodied in systems that include a long wave infrared sensor; anear infrared sensor; a data processing apparatus; and a data storagedevice storing instructions executable by the data processing apparatusthat upon execution by the data processing apparatus cause the dataprocessing apparatus to perform operations including: obtaining a longwave infrared image from the long wave infrared sensor detect an objectin the long wave infrared image, identifying a region of interestassociated with the object, adjusting a control parameter of the nearinfrared sensor based on data associated with the region of interest,obtaining a near infrared image captured using the adjusted controlparameter of the near infrared sensor, and determining a classificationof the object based on data of the near infrared image associated withthe region of interest.

In a third aspect, the subject matter described in this specificationcan be embodied in methods that include obtaining a long wave infraredimage from a long wave infrared sensor; detecting an object in the longwave infrared image; identifying a region of interest associated withthe object; adjusting a control parameter of a near infrared sensorbased on data associated with the region of interest; obtaining a nearinfrared image captured using the adjusted control parameter of the nearinfrared sensor; and determining a classification of the object based ondata of the near infrared image associated with the region of interest.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosure is best understood from the following detaileddescription when read in conjunction with the accompanying drawings. Itis emphasized that, according to common practice, the various featuresof the drawings are not to-scale. On the contrary, the dimensions of thevarious features are arbitrarily expanded or reduced for clarity.

FIG. 1 is a flowchart of an example of a process for multi-modal sensingfor nighttime autonomous object detection and recognition.

FIG. 2 is a flowchart of an example of a process for adjusting controlparameters for a region of interest.

FIG. 3A is a flowchart of an example of a process for determining aclassification of an object.

FIG. 3B is a flowchart of an example of a process for training a machinelearning unit for classification of objects.

FIG. 4 is a block diagram of an example of a vehicle configured formulti-modal sensing for nighttime autonomous object detection andrecognition.

FIG. 5 is a block diagram of an example of a hardware configuration fora vehicle controller.

FIG. 6 is a block diagram of an example of a hardware configuration of acomputing device.

FIG. 7 is a diagram of an example of overlapping fields of view formultiple sensors of different types mounted on a vehicle.

DETAILED DESCRIPTION

Nighttime or low-light environments present challenges for automatedvehicle control systems. For example, the illumination level provided byheadlights on a vehicle at night may be limited by laws or regulations,which may in turn limit the effective range of a visible spectrum sensor(e.g., a camera) used for detecting objects in or near the path of thevehicle. Having a limited effective range (e.g., about 60 meters) fordetecting and or classifying objects can reduce safety and/or reduce thespeed at which the vehicle can travel safely.

A combination of multiple complimentary image sensing technologies maybe employed to address the challenges of nighttime or low-lightenvironment object detection and classification. For example, there maybe looser or no restrictions on the illumination level of a nearinfrared illuminator mounted on a vehicle. A near infrared sensor with anear infrared illuminator can be configured to capture high resolutionimage information about objects in or near a path of the vehicle out toa significantly longer range (e.g., 200 meters) from the vehicle. Thismay enable earlier detection and classification of objects as thevehicle moves and improve safety and/or maximum speed. Near infraredilluminators may project near infrared light in a relatively narrowfield of view (e.g., a 30-degree cone).

Although their range may be relatively limited, visible spectrum sensorscan provide high resolution image data in multiple color channels (e.g.,red, green, and blue). Visible spectrum sensors also may provide a widerfield of view (e.g., a 120-degree field of view) of the path in front ofa vehicle.

Long wave infrared sensors capture naturally occurring thermal radiationfrom objects in the environment around a vehicle and therefore do notrely on an illuminator. The effective range of a long wave infraredsensor may be limited by the sensor resolution and the resolutionrequirements for object detection and/or classification. A long waveinfrared sensor, which may include an array of component sensors, mayprovide a wide field of view around the vehicle (e.g., a 180-degreefield of view). Long wave infrared sensors may provide images of objectsin the environment that are of relatively low resolution.

In some implementations, objects detected based on low resolution imagedata from a long wave infrared sensor are classified by adjustingcontrol parameters for other sensing modalities and/or image processingresources to focus computer vision resources of the vehicle on a regionof interest associated with the detected objects. For example, anintegration time, an aperture size, a filter, or a gain for a sensor(e.g., a near infrared sensor or a visible spectrum sensor) may beadjusted to enhance a portion of a captured image associated with aregion of interest. For example, a power level or a field of view for anilluminator (e.g., a near infrared illuminator or a visible spectrumilluminator) may be adjusted to enhance a portion of a captured imageassociated with a region of interest. For example, a computationalcontrol parameter (e.g., a resolution used for image processing or acount of image processing passes) may be adjusted and applied to animage portion associated with a region of interest.

The techniques described herein may provide improvements over priorcomputer vision systems for automated vehicles. Some implementations mayincrease the effective range at which objects in or near the path of avehicle may be detected and classified. Some implementations may moreaccurately classify objects in a low-light environment. Safety of anautomated vehicle control system may be improved and/or the maximum safespeed in low-light environments may be increased.

FIG. 1 is a flowchart of an example of a process 100 for multi-modalsensing for nighttime autonomous object detection and recognition. Theprocess 100 includes obtaining 110 a long wave infrared image from along wave infrared sensor; detecting 120 an object in the long waveinfrared image; identifying 130 a region of interest associated with theobject; adjusting 140 one or more control parameters based on dataassociated with the region of interest; obtaining 150 a near infraredimage captured using the adjusted control parameter(s) for a nearinfrared sensor; obtaining 160 a visible spectrum image captured usingthe adjusted control parameter(s) for a visible spectrum sensor;determining 170 a classification of the object based on data of the nearinfrared image and/or the visible spectrum image associated with theregion of interest; determining 180 a motion plan based on theclassification of the object; and outputting 190 commands to actuatorsto maneuver a vehicle. For example, the process 100 may be implementedby the automated controller 450 of FIG. 4. For example, the process 100may be implemented by the vehicle controller 500 of FIG. 5. For example,the process 100 may be implemented by the computing device 600 of FIG.6.

The process 100 includes obtaining 110 a long wave infrared image from along wave infrared sensor. For example, the long wave infrared sensormay detect electromagnetic radiation in a spectral range correspondingto thermal radiation (e.g., wavelengths of 8 micrometers to 15micrometers). The long wave infrared image may include a wide field ofview (e.g., 180 degrees). The long wave infrared image may offer lowresolution information about objects in space out to a long range fromthe sensor. For example, the long wave infrared image may be obtained110 via the long wave infrared sensor 436 of FIG. 4. For example, thelong wave infrared image may be obtained 110 via the sensor interface530 of FIG. 5. For example, the long wave infrared image may be obtained110 via the wireless interface 630 of FIG. 6.

The process 100 includes detecting 120 an object in the long waveinfrared image. For example, one or more objects may be detected 120 byidentifying clusters of pixels reflecting thermal radiation from anobject (e.g., a person, an animal, or a vehicle) that is greater thanits surroundings in the space depicted in the long wave infrared image.For example, one or more objects may be detected 120 in the long waveinfrared image using an image segmentation routine (e.g., implementingFelzenszwalb segmentation) to identify clusters of pixels in the longwave infrared image associated with an object appearing within the fieldof view of the long wave infrared image. For example, an object may bedetected 120 using the object detector 460 of FIG. 4.

The process 100 includes identifying 130 a region of interest associatedwith the object. The region of interest may be specified in coordinatesystem common to multiple sensors (e.g., a vehicle coordinate system).For example, a specification of the region of interest may include aview angle. The region of interest may correspond to image portions(e.g., blocks of pixels) in images from multiple respective imagesensors. The region of interest may be mapped to portions of images frommultiple sensors using a bundle adjustment algorithm (e.g., using theSLAM (Simultaneous Localization and Mapping) algorithm). The region ofinterest may be identified 130 based on the locations of pixels of thelong wave infrared image in a cluster associated with the object. Forexample, a view angle of the region of interest may be directed at acenter on the cluster of pixels. A size of the region of interest may bedetermined based on a size of a cluster of pixels associated with theobject. The identified 130 region of interest may correspond to imageportions of images from additional sensors (e.g., a near infrared imageand/or a visible spectrum image), which may be analyzed to determineadditional information about the detected 120 object.

The process 100 includes adjusting 140 one or more control parametersfor the region of interest to enhance the capture and/or analysis ofdata from additional sensors in the region of interest. For example, acontrol parameter of the near infrared sensor may be adjusted 140 basedon data associated with the region of interest. For example, anintegration time or exposure time for pixels in the region of interestmay be adjusted 140 to enhance contrast of a near infrared image withinthe region of interest. For example, an aperture size for the nearinfrared sensor may be adjusted 140 to enhance a near infrared imagewithin the region of interest. For example, a filter for the nearinfrared sensor may be adjusted 140 (e.g., selected) to enhance a nearinfrared image within the region of interest by selecting an appropriatespectral range for the object (e.g., based on an initial classificationof the object). For example, a gain (e.g., an electronic amplifier gain)for the near infrared sensor may be adjusted 140 to enhance a nearinfrared image within the region of interest. For example, a controlparameter (e.g., an integration time, an aperture size, a filterselection, and/or an amplifier gain) of a visible spectrum sensor may beadjusted 140 based on data associated with the region of interest. Forexample, a control parameter (e.g., a power level and/or a field ofview) of a visible spectrum illuminator may be adjusted 140 based ondata associated with the region of interest. For example, a controlparameter (e.g., a power level and/or a field of view) of a nearinfrared illuminator may be adjusted 140 based on data associated withthe region of interest. For example, a computational control parameter(e.g., a resolution for signal processing pass or a count of signalprocessing passes) may be adjusted 140 based on data associated with theregion of interest. In some implementations, control parameters formultiple sensors and image processing routines are adjusted 140 toprovide more information about the region of interest. For example, theprocess 200 of FIG. 2 may be implemented to adjust 140 controlparameters for the region of interest.

The process 100 includes obtaining 150 a near infrared image capturedusing an adjusted 140 control parameter of the near infrared sensor. Forexample, the near infrared sensor may detect electromagnetic radiationin a spectral range just below the visible range (e.g., wavelengths of0.75 micrometers to 1.4 micrometers). A near infrared illuminator may beused to generate light in this spectral range that is reflected offobjects in the space and detected by the near infrared sensor. The nearinfrared image may be captured using an adjusted 140 near infraredilluminator control parameter. The adjusted 140 control parameter mayinclude, for example, an integration time, a filter selection, anaperture size selection, and/or a gain. In some implementations, theadjusted 140 control parameter(s) may include control parameter(s)(e.g., a power level and/or a field of view) for a near infraredilluminator. The near infrared image may include a narrow field of view(e.g., 30 degrees). The near infrared image may have a higher resolutionthan the long wave infrared image and include information about objectsin space out to a long range (e.g., 200 meters) from the near infraredsensor. In some implementations, the near infrared illuminator includesan array of illuminators pointed in different directions from a vehicle(e.g., three illuminator components with respective 30 degree fields ofview that collectively span a 90 degree field of view) and near infraredilluminator components pointed off the path of the vehicle may havetheir power level modulated to low power or off (e.g., to save power)when no objects are detected within its respective field of view using along wave infrared sensor and modulated to on or a high power level whenan object is detected within its respective field of view using a longwave infrared sensor. For example, the near infrared image may beobtained 150 via the near infrared sensor 434 of FIG. 4. For example,the near infrared image may be obtained 150 via the sensor interface 530of FIG. 5. For example, the near infrared image may be obtained 150 viathe wireless interface 630 of FIG. 6.

The process 100 includes obtaining 160 a visible spectrum image from avisible spectrum sensor that is captured using the adjusted controlparameter of the visible spectrum sensor. For example, the visiblespectrum sensor may detect electromagnetic radiation in a spectral rangethat is visible to humans (e.g., wavelengths of 400 nanometers to 700nanometers). The visible spectrum sensor may capture light in multiplespectral subranges corresponding to different colors (e.g., red, green,and blue) and the visible spectrum image may include multiple colorchannels (e.g., RGB or YCrCb). A visible spectrum illuminator (e.g., aheadlight on a vehicle) may be used to generate light in this spectralrange that is reflected off objects in the space and detected by thevisible spectrum sensor. The visible spectrum image may be capturedusing an adjusted 140 visible spectrum illuminator control parameter.The adjusted 140 control parameter for the visible spectrum sensor mayinclude, for example, an integration time, a filter selection, anaperture size selection, and/or a gain. In some implementations, theadjusted 140 control parameter(s) may include control parameter(s)(e.g., a power level and/or a field of view) for a visible spectrumilluminator. The visible spectrum image may include a field of view(e.g., 120 degrees or 180 degrees). The visible spectrum image may havea higher resolution than the long wave infrared image and includeinformation about objects in the space out to a short range (e.g., 60meters) from the visible spectrum sensor. For example, the visiblespectrum image may be obtained 160 via the visible spectrum sensor 432of FIG. 4. For example, the visible spectrum image may be obtained 160via the sensor interface 530 of FIG. 5. For example, the visiblespectrum image may be obtained 160 via the wireless interface 630 ofFIG. 6.

The process 100 includes determining 170 a classification (e.g., as aperson, an animal, a vehicle, a barrier, a building, a traffic sign,static, dynamic, etc.) of the object based on data from one or moresensors associated with the region of interest. For example, aclassification of the object may be determined 170 based on data of thenear infrared image associated with the region of interest. For example,the classification of the object may be determined 170 based on data ofthe visible spectrum image associated with the region of interest. Imagedata (e.g., from the visible spectrum image, from the near infraredimage, and/or from the long wave infrared image) for the region ofinterest may pre-processed and/or input to a machine learning unit(e.g., including a convolutional neural network) that outputs aclassification of the object appearing in the region of interest. Insome implementations, a classification of the object is determined usingan adjusted 140 computational control parameter (e.g., a resolution, astride, or a count of pre-processing passes). For example, the process300 of FIG. 3A may be implemented to determine 170 a classification ofthe object. For example, the object classifier 470 may be used todetermine 170 a classification of the object. In some implementations(not explicitly shown in FIG. 1), the control parameter(s) used toobtain and process sensor data for the region of interest may beiteratively adjusted 140 to determine 170 a classification of the objectappearing in the region of interest.

The process 100 includes determining 180 a motion plan based on theclassification of the object. For example, an object classification maybe used by an object tracker to generate object tracking data includingprojected paths for dynamic objects, which may be used to determine 180a motion plan for collision avoidance or passing. For example, themotion plan may be determined 180 by the automated controller 450 ofFIG. 4.

The process 100 includes outputting 190 commands to actuators tomaneuver a vehicle. The commands may be based on the motion plan. Forexample, commands may be output 190 to a power source and transmissionsystem (e.g., the power source and transmission system 422), a steeringsystem (e.g., the steering system 424), and/or a braking system (e.g.,the braking system 426). For example, the commands may be output 190 bythe automated controller 450, the vehicle controller 500, or thecomputing device 600. For example, the commands may be output 190 viathe controller interface 540, or the wireless interface 630. Forexample, maneuvering the vehicle may include accelerating, turning,and/or stopping.

FIG. 2 is a flowchart of an example of a process 200 for adjustingcontrol parameters for a region of interest. The process 200 includesadjusting 210 one or more control parameters of a near infrared sensor;adjusting 220 one or more control parameters of a near infraredilluminator; adjusting 230 one or more control parameters of a visiblespectrum sensor; adjusting 240 one or more control parameters of avisible spectrum illuminator; and adjusting 250 one or more controlparameters of a computational routine for processing image data for theregion of interest. For example, the process 200 may be implemented bythe automated controller 450 of FIG. 4. For example, the process 200 maybe implemented by the vehicle controller 500 of FIG. 5. For example, theprocess 200 may be implemented by the computing device 600 of FIG. 6.

The process 200 includes adjusting 210 one or more control parameters ofa near infrared sensor. For example, an adjusted 210 control parameterof the near infrared sensor may be an integration time. The integrationtime may be a duration of time during which an image sensing element ofthe near infrared sensor collects photons prior to sampling for an imagecapture. For example, an adjusted 210 control parameter of the nearinfrared sensor may be an aperture size. An aperture size may beadjusted 210 mechanically by expanding or contracting an aperture incover of the image sensor or by swapping in a cover with a differentaperture size. For example, an adjusted 210 control parameter of thenear infrared sensor may be a filter selection. A filter selection maycause an optical filter (e.g., made of glass or plastic) to bemechanically moved into or out of position over a sensing element of thenear infrared sensor. For example, an adjusted 210 control parameter ofthe near infrared sensor may be an amplification gain (e.g., anelectronic amplifier gain).

The process 200 includes adjusting 220 a near infrared illuminatorcontrol parameter based on data associated with the region of interest.For example, an adjusted 220 near infrared illuminator control parametermay be a brightness. The brightness of the illuminator may beproportional to power level and/or an illumination level. For example,an adjusted 220 near infrared illuminator control parameter is field ofillumination. The field of illumination may be adjusted 220 by changinga lens covering the illuminator.

The process 200 includes adjusting 230 a control parameter of thevisible spectrum sensor based on data associated with the region ofinterest. For example, an adjusted 230 control parameter of the visiblespectrum sensor may be an integration time. The integration time may bea duration of time during which an image sensing element of the visiblespectrum sensor collects photons prior to sampling for an image capture.For example, an adjusted 230 control parameter of the visible spectrumsensor may be an aperture size. An aperture size may be adjusted 230mechanically by expanding or contracting an aperture in cover of theimage sensor or by swapping in a cover with a different aperture size.For example, an adjusted 230 control parameter of the visible spectrumsensor may be a filter selection. A filter selection may cause anoptical filter (e.g., made of glass or plastic) to be mechanically movedinto or out of position over a sensing element of the visible spectrumsensor. For example, an adjusted 230 control parameter of the visiblespectrum sensor may be an amplification gain (e.g., an electronicamplifier gain).

The process 200 includes adjusting 240 a visible spectrum illuminatorcontrol parameter based on data associated with the region of interest.For example, an adjusted 240 visible spectrum illuminator controlparameter may be a brightness. The brightness of the illuminator may beproportional to power level and/or an illumination level. For example,an adjusted 240 visible spectrum illuminator control parameter is fieldof illumination. The field of illumination may be adjusted 240 bychanging a lens covering the illuminator.

The process 200 includes adjusting 250 a computational control parameterbased on data associated with the region of interest. For example, acomputational control parameter may specify a resolution at which imagedata from sensors (e.g., a near infrared sensor or a visible spectrumsensor) will be image processed to extract information relating to theobject. For example, a computational control parameter may specify astride size that will be used by a convolutional layer of convolutionalneural network to process image data from sensors (e.g., a near infraredsensor or a visible spectrum sensor) to extract information relating tothe object. For example, a computational control parameter may specify acount of pre-processing passes that will be applied to image data fromsensors (e.g., a near infrared sensor or a visible spectrum sensor) toextract information relating to the object.

FIG. 3A is a flowchart of an example of a process 300 for determining aclassification of an object. The process 300 includes pre-processing 310images from sensors to extract features for a region of interest; fusing320 features from multiple sensors for the region of interest; andinputting 330 the features to a machine learning unit to determine aclassification of an object appearing in the region of interest. Forexample, the process 300 may be implemented by the automated controller450 of FIG. 4. For example, the process 300 may be implemented by thevehicle controller 500 of FIG. 5. For example, the process 300 may beimplemented by the computing device 600 of FIG. 6.

The process 300 includes pre-processing 310 images from sensors toextract features for a region of interest. For example, pre-processing310 a Bayer filtered visible spectrum image may include demosaicing thevisible spectrum image. For example, pre-processing 310 a visiblespectrum image, a near infrared image, and/or a long wave infrared imagemay include applying noise reduction filtering (e.g., spatial noisereduction filtering and/or temporal noise reduction filtering). In someimplementations, pre-processing 310 an image from one of the sensorsincludes applying a transformation (e.g., a discrete cosine transform ora wavelet transform) or a matched filter to extract features (e.g.,frequency or scale features) from an image portion corresponding to theregion of interest. In some implementations, pixel values for an imageportion corresponding to the region of interest extracted as features.

The process 300 includes fusing 320 features from multiple sensors forthe region of interest. In some implementations, features extractedimages captured with different sensors may be resampled to facilitatethe fusing of image channels taken from multiple source images (e.g., avisible spectrum image and a near infrared image) at common resolutionfor analysis with a machine learning unit (e.g., a convolutional neuralnetwork).

The process 300 includes inputting 330 the features to a machinelearning unit to determine a classification (e.g., as a person, ananimal, a vehicle, a barrier, a building, a traffic sign, static,dynamic, etc.) of an object appearing in the region of interest. Forexample, the machine learning unit may include a convolutional neuralnetwork, a support vector machine, or a Bayesian network. For example,the machine learning unit may be trained using the process 350 of FIG.3B.

FIG. 3B is a flowchart of an example of a process 350 for training amachine learning unit for classification of objects. The process 350includes obtaining 352 training data; labeling 354 training data withground truth labels; and training 356 a machine learning unit using thetraining data and the ground truth labels. For example, the machinelearning unit may be a convolutional neural network and it may betrained 356 using a back-propagation algorithm. For example, the process350 may be implemented by the automated controller 450 of FIG. 4. Forexample, the process 350 may be implemented by the vehicle controller500 of FIG. 5. For example, the process 350 may be implemented by thecomputing device 600 of FIG. 6.

FIG. 4 is a block diagram of an example of a vehicle configured formulti-modal sensing for nighttime autonomous object detection andrecognition. The vehicle 400 includes a vehicle body 410 that containsor is attached to the other systems and components of the vehicle 400.The vehicle 400 includes wheels 420 that are capable of serving as aninterface between the vehicle 400 and a road. The wheels 420 providecontrol surfaces that may be used to guide the vehicle along paths on aroad. The vehicle 400 includes actuators operable to cause motion of thevehicle body 410. The actuators include a power and transmission system422, a steering system 424, and a braking system 426. The vehicle 400includes a sensor group 430 for sensing an environment near the vehicle400. The vehicle 400 includes an automated controller 450 configured tomaneuver the vehicle, based on sensor data from the sensor group 430, bysending control signals to the actuators (e.g., the power andtransmission system 422, the steering system 424, and/or the brakingsystem 426). For example, the vehicle 400 may use the automatedcontroller 450 to implement the process 100 of FIG. 1.

The vehicle 400 includes a power source (e.g., a combustion engineand/or a battery) connected to the wheels via a transmission system 422capable of spinning the wheels to accelerate the vehicle along a road.The vehicle 400 includes a steering system 424 capable of turning thewheels 420 in relation to the vehicle body 410 to direct the motion ofthe vehicle, e.g., by controlling the yaw angle and angular velocity orpath curvature of the vehicle.

The vehicle 400 includes a sensor group 430, configured to detect otherobjects near the vehicle. The sensor group 430 may include a variety ofsensors including a visible spectrum sensor 432, a near infrared sensor434, a long wave infrared sensor 436, and/or additional sensors (notshown), such as an accelerometer, a gyroscope, a magnetometer, anodometer, a global positioning system receiver, a lidar sensor, a radarsensor, etc. The sensor group 430 may also include illuminators, such asthe visible spectrum illuminator 438 and the near infrared illuminator440, that provide light that is reflected of objects in the environmentto facilitate detection with the corresponding image sensors. Theilluminators of the vehicle may be particularly useful when operating atnight.

The sensor group 430 includes a visible spectrum sensor 432 (e.g., acamera or an array of cameras) configured to capture visible spectrumimages of objects in a space near the vehicle. For example, the visiblespectrum sensor 432 may detect electromagnetic radiation in a spectralrange that is visible to humans (e.g., wavelengths of 400 nanometers to700 nanometers). The visible spectrum sensor 432 may capture light inmultiple spectral subranges corresponding to different colors (e.g.,red, green, and blue) and a visible spectrum image output by the visiblespectrum sensor 432 may include multiple color channels (e.g., RGB orYCrCb). The visible spectrum sensor 432 may include a color filter array(e.g., a Bayer filter) for capturing a multi-channel visible spectrumimage. In some implementations, the visible spectrum sensor is singlechannel (e.g., with a single filter for all sensor elements) and outputsblack and white images. The visible spectrum sensor 432 may beconfigured to enhance the quality of a captured image in a region ofinterest by adjusting one or more control parameters (e.g., integrationtime, a filter selection, an aperture size selection, and/or a gain) forthe visible spectrum sensor. The visible spectrum image may include afield of view (e.g., 120 degrees or 180 degrees). For example, thevisible spectrum sensor 432 may provide the visible field of view 720described in FIG. 7. The visible spectrum image may have a higherresolution than the long wave infrared image and include informationabout objects in the space out to a short range (e.g., 60 meters) fromthe visible spectrum sensor. The range may be limited, particularly atnight, by the illumination level provided by the visible spectrumilluminator 438.

The sensor group 430 includes a near infrared sensor 434 configured tocapture near infrared images of objects in a space near the vehicle. Forexample, the near infrared sensor 434 may detect electromagneticradiation in a spectral range just below the visible range (e.g.,wavelengths of 0.75 micrometers to 1.4 micrometers). The near infraredsensor 434 may be configured to enhance the quality of a captured imagein a region of interest by adjusting one or more control parameters(e.g., integration time, a filter selection, an aperture size selection,and/or a gain) for the near infrared sensor 434. The near infraredsensor 434 may provide a narrow field of view (e.g., 30 degrees). Forexample, the near infrared sensor 434 may provide the NIR field of view740 described in FIG. 7. The near infrared image may have a higherresolution than the long wave infrared image and include informationabout objects in space out to a long range (e.g., 200 meters) from thenear infrared sensor. The range may be limited by the illumination levelprovided by the near infrared illuminator 440.

The sensor group 430 includes a long wave infrared sensor 436 configuredto capture long wave infrared images of objects in a space near thevehicle. For example, the long wave infrared sensor 436 may detectelectromagnetic radiation in a spectral range corresponding to thermalradiation (e.g., wavelengths of 8 micrometers to 15 micrometers). Thelong wave infrared sensor 436 may provide a wide field of view (e.g.,180 degrees). For example, the long wave infrared sensor 436 may providethe LWIR field of view 730 described in FIG. 7. A long wave infraredimage from the long wave infrared sensor 436 may offer low resolutioninformation about objects in space out to a long range from the sensor.

The sensor group 430 includes a visible spectrum illuminator 438 (e.g.,a headlight on a vehicle) configured to project visible light from thevehicle onto objects near the vehicle 400 to facilitate capture ofvisible spectrum images. The visible spectrum illuminator 438 mayinclude one or more lens that direct light from the visible spectrumilluminator 438 and determine a field of view for the visible spectrumilluminator 438. For example, the visible spectrum illuminator 438 canbe used to generate light in this spectral range that is reflected offobjects in the space and detected by the visible spectrum sensor 432. Avisible spectrum image may be captured using one or more adjustedcontrol parameters (e.g., a power level and/or a field of view) of thevisible spectrum illuminator 438. For example, the illumination levelfor the visible spectrum illuminator 438 may be limited by laws orregulations and/or a power budget for the vehicle 400.

The sensor group 430 includes a near infrared illuminator 440 configuredto project near infrared light from the vehicle onto objects near thevehicle 400 to facilitate capture of near infrared images. For example,the infrared illuminator 440 can be used to generate light in thisspectral range that is reflected off objects in the space and detectedby the near infrared sensor 434. The infrared illuminator 440 mayinclude one or more lens that direct light from the infrared illuminator440 and determine a field of view for the infrared illuminator 440. Anear infrared image may be captured using one or more adjusted controlparameters (e.g., a power level and/or a field of view) of the infraredilluminator 440. For example, the illumination level for the infraredilluminator 440 may be limited by laws or regulations and/or a powerbudget for the vehicle 400.

The vehicle 400 includes an automated controller 450 that is configuredto receive data from the sensor group 430 and possibly other sources(e.g., a vehicle passenger/operator control interface) and process thedata to implement automated control of the motion of the vehicle 400 bysending control signals to actuators (e.g., the Power source &transmission system 422, the steering system 424, and the braking system426) that actuate these commands via the wheels 420 to maneuver thevehicle 400. The automated controller 450 may be configured to sendcontrol signals to the sensor group 430 and receive sensor data from thesensor group 430. For example, the automated controller 450 may sendadjusted control parameters to the sensor group 430 that control theconfiguration of sensors and/or illuminators for sensor data capturethat is tailored to enhance a region of interest associated with adetected object. In some implementations, the automated controller 450is configured to detect and classify objects at night in a space nearthe vehicle to inform control of the vehicle 400. For example, theautomated controller 450 may be configured to implement process 100 asdescribed in relation to FIG. 1. The automated controller 450 mayinclude specialized data processing and control hardware and/or softwarerunning on a data processing apparatus with additional capabilities. Forexample, the automated controller 450 may be implemented using thevehicle controller 500 of FIG. 5.

The automated controller 450 includes or interfaces with an objectdetector 460 that is configured to process. For example, the objectdetector 460 may detect one or more objects by identifying clusters ofpixels reflecting thermal radiation from an object (e.g., a person, ananimal, or a vehicle) that is greater than its surroundings in the spacedepicted in a long wave infrared image. For example, one or more objectsmay be detected in the long wave infrared image using an imagesegmentation algorithm (e.g., the Felzenszwalb segmentation algorithm)to identify clusters of pixels in the long wave infrared imageassociated with an object appearing within the field of view of the longwave infrared image. The object detector 460 may include specializeddata processing and control hardware and/or software running on a dataprocessing apparatus with additional capabilities.

The automated controller 450 includes or interfaces with an objectclassifier 470 that is configured to classify objects in a region ofinterest based on image data from sensors corresponding to the region ofinterest. The automated controller 450 may pass image data for theregion of interest from multiple sensors (e.g., the visible spectrumsensor 432, the near infrared sensor 434, and/or the long wave infraredsensor 436) to the object classifier 470 determine a classification(e.g., as a person, an animal, a vehicle, a barrier, a building, atraffic sign, static, dynamic, etc.) for an object appearing in theregion of interest. For example, the object classifier 470 may include aconvolutional neural network. In some implementations, a classificationof an object is determined using a computational control parameter(e.g., a resolution, a stride, or a count of pre-processing passes) thathas been adjusted based on data for the region of interest (e.g., dataspecifying a location and/or a size of the region of interest). Forexample, the object classifier 470 may implement the process 300 of FIG.3A to classify an object. The object classifier 470 may implement theprocess 350 of FIG. 3B to train a machine learning component of theobject classifier 470. The object classifier 470 may include specializeddata processing and control hardware and/or software running on a dataprocessing apparatus with additional capabilities.

The automated controller 450 includes or interfaces with a map localizer480 that is configured to fuse data from multiple sensors of the vehicle400 and update a navigation map based on local sensor data. In someimplementations, the map localizer may implement a bundle adjustmentalgorithm (e.g., the SLAM algorithm). The automated controller 450 maypass a classification of an object in a region of interest to the maplocalizer 480 to facilitate updating a navigation map. The map localizer480 may include specialized data processing and control hardware and/orsoftware running on a data processing apparatus with additionalcapabilities.

FIG. 5 is a block diagram of an example of a hardware configuration fora vehicle controller 500. The hardware configuration may include a dataprocessing apparatus 510, a data storage device 520, a sensor interface530, a controller interface 540, and an interconnect 550 through whichthe data processing apparatus 510 may access the other components. Forexample, the vehicle controller 500 may be configured to implement theprocess 100 of FIG. 1.

The data processing apparatus 510 is operable to execute instructionsthat have been stored in a data storage device 520. In someimplementations, the data processing apparatus 510 is a processor withrandom access memory for temporarily storing instructions read from thedata storage device 520 while the instructions are being executed. Thedata processing apparatus 510 may include single or multiple processorseach having single or multiple processing cores. Alternatively, the dataprocessing apparatus 510 may include another type of device, or multipledevices, capable of manipulating or processing data. For example, thedata storage device 520 may be a non-volatile information storage devicesuch as a hard drive, a solid-state drive, a read-only memory device(ROM), an optical disc, a magnetic disc, or any other suitable type ofstorage device such as a non-transitory computer readable memory. Thedata storage device 520 may include another type of device, or multipledevices, capable of storing data for retrieval or processing by the dataprocessing apparatus 510. For example, the data storage device 520 canbe distributed across multiple machines or devices such as network-basedmemory or memory in multiple machines performing operations that can bedescribed herein as being performed using a single computing device forease of explanation. The data processing apparatus 510 may access andmanipulate data in stored in the data storage device 520 viainterconnect 550. For example, the data storage device 520 may storeinstructions executable by the data processing apparatus 510 that uponexecution by the data processing apparatus 510 cause the data processingapparatus 510 to perform operations (e.g., operations that implement theprocess 100 of FIG. 1).

The sensor interface 530 may be configured to control and/or receiveimage data (e.g., a long wave infrared image, a near infrared image,and/or a visible spectrum image) from one or more sensors (e.g., thevisible spectrum sensor 432, the near infrared sensor 434, and/or thelong wave infrared sensor 436). In some implementations, the sensorinterface 530 may implement a serial port protocol (e.g., I2C or SPI)for communications with one or more sensor devices over conductors. Insome implementations, the sensor interface 530 may include a wirelessinterface for communicating with one or more sensor groups vialow-power, short-range communications (e.g., using a vehicle areanetwork protocol).

The controller interface 540 allows input and output of information toother systems within a vehicle to facilitate automated control of thevehicle. For example, the controller interface 540 may include serialports (e.g., RS-232 or USB) used to issue control signals to actuatorsin the vehicle (e.g., the power source and transmission system 422, thesteering system 424, and the braking system 426) and to receive sensordata from a sensor group (e.g., the sensor group 430. For example, theinterconnect 550 may be a system bus, or a wired or wireless network(e.g., a vehicle area network).

FIG. 6 is a block diagram of an example of a hardware configuration of acomputing device 600. The hardware configuration may include a dataprocessing apparatus 610, a data storage device 620, wireless interface630, a user interface 640, and an interconnect 650 through which thedata processing apparatus 610 may access the other components. Thecomputing device may be configured to detect and classify objects atnight based on image data from multiple sensors. For example, thecomputing device 600 may be configured to implement the process 100 ofFIG. 1.

The data processing apparatus 610 is operable to execute instructionsthat have been stored in a data storage device 620. In someimplementations, the data processing apparatus 610 is a processor withrandom access memory for temporarily storing instructions read from thedata storage device 620 while the instructions are being executed. Thedata processing apparatus 610 may include single or multiple processorseach having single or multiple processing cores. Alternatively, the dataprocessing apparatus 610 may include another type of device, or multipledevices, capable of manipulating or processing data. For example, thedata storage device 620 may be a non-volatile information storage devicesuch as a hard drive, a solid-state drive, a read-only memory device(ROM), an optical disc, a magnetic disc, or any other suitable type ofstorage device such as a non-transitory computer readable memory. Thedata storage device 620 may include another type of device, or multipledevices, capable of storing data for retrieval or processing by the dataprocessing apparatus 610. For example, the data storage device 620 canbe distributed across multiple machines or devices such as network-basedmemory or memory in multiple machines performing operations that can bedescribed herein as being performed using a single computing device forease of explanation. The data processing apparatus 610 may access andmanipulate data in stored in the data storage device 620 viainterconnect 650. For example, the data storage device 620 may storeinstructions executable by the data processing apparatus 610 that uponexecution by the data processing apparatus 610 cause the data processingapparatus 610 to perform operations (e.g., operations that implement theprocess 100 of FIG. 1).

The wireless interface 630 facilitates communication with other devices,for example, a vehicle (e.g., the vehicle 400). For example, wirelessinterface 630 may facilitate communication via a vehicle Wi-Fi networkwith a vehicle controller (e.g., the vehicle controller 500 of FIG. 5).For example, wireless interface 630 may facilitate communication via aWiMAX network with a vehicle at a remote location.

The user interface 640 allows input and output of information from/to auser. In some implementations, the user interface 640 can include adisplay, which can be a liquid crystal display (LCD), a cathode-ray tube(CRT), a light emitting diode (LED) display (e.g., an OLED display), orother suitable display. For example, the user interface 640 may includea touchscreen. For example, the user interface 640 may include ahead-mounted display (e.g., virtual reality goggles or augmented realityglasses). For example, the user interface 640 may include a positionalinput device, such as a mouse, touchpad, touchscreen, or the like; akeyboard; or other suitable human or machine interface devices. Forexample, the interconnect 650 may be a system bus, or a wired orwireless network (e.g., a vehicle area network).

FIG. 7 is a diagram of an example of overlapping fields of view 700 formultiple sensors of different types mounted on a vehicle 710. Thevehicle 710 includes a visible spectrum sensor and a visible spectrumilluminator that together provide a corresponding visible field of view720 that spans about 120 degrees and extends 60 meters in front of thevehicle. The visible spectrum images from this visible spectrum sensormay include three color channels (e.g., red, green, and blue). The rangeof the visible spectrum illuminator and sensor may be limited by lawsand regulations intended to prevent blinding oncoming traffic and lightpollution. Thus, the visible spectrum sensor may provide high resolutioncolor images at relatively short range.

The vehicle 710 includes a long wave infrared sensor that provides acorresponding LWIR field of view 730 that spans 180 degrees with aneffective range (at reasonable resolution) that extends 120 meters infront of the vehicle. Thus, the long wave infrared sensor may providelow resolution images at relatively moderate to long range.

The vehicle 710 includes a near infrared sensor and a near infraredilluminator that together provide a corresponding NIR field of view 740that spans about 30 degrees and extends 200 meters in front of thevehicle. Thus, the near infrared sensor may provide high resolutionmonochromatic images at relatively long range.

These overlapping fields of view 700 may provide complimentaryinformation that can be used to facilitate robust detection andclassification of objects at night and in low ambient lightenvironments. For example, low resolution information from a long waveinfrared image may be used to detect objects and direct illumination(e.g., near infrared and/or visible spectrum light) and focus imageprocessing resources of higher resolution modalities at the detectedobject to facilitate classification of the object.

While the disclosure has been described in connection with certainembodiments, it is to be understood that the disclosure is not to belimited to the disclosed embodiments but, on the contrary, is intendedto cover various modifications and equivalent arrangements includedwithin the scope of the appended claims, which scope is to be accordedthe broadest interpretation so as to encompass all such modificationsand equivalent structures as is permitted under the law.

What is claimed is:
 1. A system, comprising: a long wave infraredsensor; a near infrared sensor; a data processing apparatus; and a datastorage device storing instructions executable by the data processingapparatus that upon execution by the data processing apparatus cause thedata processing apparatus to perform operations comprising: obtaining along wave infrared image from the long wave infrared sensor, detectingan object in the long wave infrared image, identifying a region ofinterest associated with the object, adjusting a control parameter ofthe near infrared sensor based on data associated with the region ofinterest, obtaining a near infrared image captured using the adjustedcontrol parameter of the near infrared sensor, and determining aclassification of the object based on data of the near infrared imageassociated with the region of interest.
 2. The system of claim 1,comprising: a near infrared illuminator; wherein the operations compriseadjusting a near infrared illuminator control parameter based on dataassociated with the region of interest; and wherein the near infraredimage is captured using the adjusted near infrared illuminator controlparameter.
 3. The system of claim 2, wherein the near infraredilluminator control parameter is a brightness.
 4. The system of claim 2,wherein the near infrared illuminator control parameter is field ofillumination.
 5. The system of claim 1, wherein the control parameter ofthe near infrared sensor is an integration time.
 6. The system of claim1, wherein the control parameter of the near infrared sensor is anaperture size.
 7. The system of claim 1, wherein the control parameterof the near infrared sensor is a filter selection.
 8. The system ofclaim 1, wherein the control parameter of the near infrared sensor is anamplification gain.
 9. The system of claim 1, comprising: a visiblespectrum sensor; wherein the operations comprise adjusting a controlparameter of the visible spectrum sensor based on data associated withthe region of interest; wherein the operations comprise obtaining avisible spectrum image from the visible spectrum sensor that is capturedusing the adjusted control parameter of the visible spectrum sensor; andwherein the classification of the object is determined based on data ofthe visible spectrum image associated with the region of interest. 10.The system of claim 9, comprising: a visible spectrum illuminator;wherein the operations comprise adjusting a visible spectrum illuminatorcontrol parameter based on data associated with the region of interest;and wherein the visible spectrum image is captured using the adjustedvisible spectrum illuminator control parameter.
 11. The system of claim1, wherein the operations comprise: adjusting a computational controlparameter based on data associated with the region of interest; andwherein the classification of the object is determined using thecomputational control parameter.
 12. A method comprising: obtaining along wave infrared image from a long wave infrared sensor; detecting anobject in the long wave infrared image; identifying a region of interestassociated with the object; adjusting a control parameter of a nearinfrared sensor based on data associated with the region of interest;obtaining a near infrared image captured using the adjusted controlparameter of the near infrared sensor; and determining a classificationof the object based on data of the near infrared image associated withthe region of interest.
 13. The method of claim 12, comprising:adjusting a visible spectrum sensor control parameter based on dataassociated with the region of interest; obtaining a visible spectrumimage from the visible spectrum sensor that is captured using theadjusted control parameter of the visible spectrum sensor; and whereinthe classification of the object is determined based on data of thevisible spectrum image associated with the region of interest.
 14. Themethod of claim 13, comprising: adjusting a visible spectrum illuminatorcontrol parameter based on data associated with the region of interest;and wherein the visible spectrum image is captured using the adjustedvisible spectrum illuminator control parameter.
 15. The method of claim12, comprising: adjusting a near infrared illuminator control parameterbased on data associated with the region of interest; and wherein thenear infrared image is captured using the adjusted near infraredilluminator control parameter.
 16. The method of claim 12, wherein thenear infrared image has a higher resolution than the long wave infraredimage.
 17. A vehicle comprising: a vehicle body; actuators operable tocause motion of the vehicle body; a long wave infrared sensor; a nearinfrared sensor; and an automated controller configured to: obtain along wave infrared image from the long wave infrared sensor, detect anobject in the long wave infrared image, identify a region of interestassociated with the object, adjust a control parameter of the nearinfrared sensor based on data associated with the region of interest,obtain a near infrared image captured using the adjusted controlparameter of the near infrared sensor, determine a classification of theobject based on data of the near infrared image associated with theregion of interest, determine a motion plan based on the classificationof the object, and output commands to the actuators to maneuver thevehicle.
 18. The vehicle of claim 17, comprising: a visible spectrumsensor; wherein the automated controller is configured to: adjust acontrol parameter of the visible spectrum sensor based on dataassociated with the region of interest, and obtain a visible spectrumimage from the visible spectrum sensor that is captured using theadjusted control parameter of the visible spectrum sensor; and whereinthe classification of the object is determined based on data of thevisible spectrum image associated with the region of interest.
 19. Thevehicle of claim 18, comprising: a visible spectrum illuminator; whereinthe automated controller is configured to adjust a visible spectrumilluminator control parameter based on data associated with the regionof interest; and wherein the visible spectrum image is captured usingthe adjusted visible spectrum illuminator control parameter.
 20. Thevehicle of claim 17, comprising: a near infrared illuminator; whereinthe automated controller is configured to adjust a near infraredilluminator control parameter based on data associated with the regionof interest; and wherein the near infrared image is captured using theadjusted near infrared illuminator control parameter.